Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support for zstd compression layer #58

Closed
wants to merge 1 commit into from
Closed

Support for zstd compression layer #58

wants to merge 1 commit into from

Conversation

utam0k
Copy link
Contributor

@utam0k utam0k commented Oct 14, 2022 •
edited
Loading

Description

Support for zstd123 compression.
zstd is said to be much faster than gzip. However, I believe zstd is still in the experimental stage.
opencontainers/image-spec#803

Moby has completed the corresponding implementation but it has not yet been released. buildkit has completed its support, as you can see from its implementation. nerdctl can be used to confirm that it works.

{
  "schemaVersion": 2,
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:a2723dd59ae9025fd33b77ae7316a77406fcd08d0077bcf5819fec5e5e49f6e6",
    "size": 3930
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:fb0b3276a519f5e7085f51c75989b287b234b3508e1524cf2cdcbc397c06ec3d",
      "size": 28574451
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
      "digest": "sha256:f3bdd52e8a0a2c581486e84c029b6e45571656416d62d9f11561714a7e202ec1",
      "size": 139
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
      "digest": "sha256:21930dd1a8bd2c1d054c0c737ff3ff21fa0ae99fc830ab44df1136c72f94352b",
      "size": 6542185
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
      "digest": "sha256:6cf4a0e124673f39bd1ea8c7ff37f22e929e3db4217a5bf5e5d165a8efc88573",
      "size": 123049956
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
      "digest": "sha256:693583e208bc9a66aaf4096d4488d38768fcf4445602de8cda06e4dbcdc4c6e7",
      "size": 65642862
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
      "digest": "sha256:92d8ef9231fc6aa6a7b4776809c54257a2aa4924e11a276a8f00fd876dcadedc",
      "size": 33833622
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
      "digest": "sha256:e0ba2d4918a6886a1e4afd68840266200e7a5bc4433bce1d6029e1f8b1f1cebf",
      "size": 774
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
      "digest": "sha256:4d37bcae4df9957c8e7fc3dcbb8b1d0d206515b9fd088e4dd7e22ee02b9cc01a",
      "size": 336616
    }
  ],
  "annotations": {
    "dazzle.gitpod.io/base-ref": "localhost:5000/dazzle:base--bd2b6e028018842142b5de640d18f91d9ac8cb22eae851087e116b678ac95f36@sha256:0d19fc62337e316b122853b1a319d500fcc5857f1fe3f4b56ff14d47971e6881"
  }
}

Related Issue(s)

gitpod-io/workspace-images#951 (comment)

How to test

$ gp await-port 5000
$ go run main.go build --context example localhost:5000/dazzle --compression zstd
$ go run main.go combine --context example localhost:5000/dazzle --all --compression zstd
$ go install github.com/csweichel/oci-tool@latest
$ oci-tool fetch raw localhost:5000/dazzle:all | jq # you can see `tar+zstd`

You can confirm that nerdctl works from your terminal by opening port 5000.

Release Notes

dazzle: Support for zstd compression

Documentation

Footnotes

  1. https://www.infoq.com/news/2022/09/amazon-gzip-zstd/ ↩

  2. https://aws.amazon.com/blogs/containers/reducing-aws-fargate-startup-times-with-zstd-compressed-container-images/ ↩

  3. https://www.slideshare.net/KoheiTokunaga/starting-up-containers-super-fast-with-lazy-pulling-of-images ↩

@utam0k utam0k self-assigned this Oct 14, 2022
@codecov
Copy link

codecov bot commented Oct 14, 2022 •
edited
Loading

Codecov Report

Merging #58 (7da8f29) into main (d889756) will increase coverage by 0.08%.
The diff coverage is 68.18%.

@@            Coverage Diff             @@
##             main      #58      +/-   ##
==========================================
+ Coverage   51.54%   51.63%   +0.08%     
==========================================
  Files           4        5       +1     
  Lines         906      920      +14     
==========================================
+ Hits          467      475       +8     
- Misses        331      337       +6     
  Partials      108      108
Impacted Files Coverage Δ
pkg/dazzle/combiner.go 24.73% <0.00%> (ø)
pkg/dazzle/compression.go 50.00% <50.00%> (ø)
pkg/dazzle/build.go 59.32% <100.00%> (+0.21%) ⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@utam0k utam0k changed the title Support for zstd compression Support for zstd compression layer Oct 14, 2022
@utam0k
Copy link
Contributor Author

utam0k commented Oct 14, 2022

@aledbf Shall we support estargz too?

@aledbf
Copy link
Member

aledbf commented Oct 14, 2022

@aledbf Shall we support estargz too?

No.

Edit: please prioritize the scheduled work. This is only a nice thing to have.

@utam0k
Copy link
Contributor Author

utam0k commented Oct 14, 2022

@aledbf Shall we support estargz too?

No.

Edit: please prioritize the scheduled work. This is only a nice thing to have.

Sure.

@utam0k utam0k mentioned this pull request Oct 14, 2022
Closed
@kylos101
Copy link
Contributor

Flipping this back to draft and removing from our project for now.

@aledbf @utam0k if you'd like to pick this up again, may I ask you to start with a RFC? This way we can visualize the involved components and overall idea.

@kylos101 kylos101 marked this pull request as draft October 26, 2022 23:34
@utam0k
Copy link
Contributor Author

utam0k commented Oct 31, 2022 •
edited
Loading

@aledbf Do you have the plan to support eStargz near future? I asked because I'm afraid that if you support it, we don't need to support zstd, right?
If you don't mind, I'd like to write an RFC about changing the compression algorithm of the image layers, eStargz or zstd.

@kylos101 kylos101 closed this Dec 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@aledbf aledbf Awaiting requested review from aledbf

@csweichel csweichel Awaiting requested review from csweichel

Assignees

@utam0k utam0k

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@utam0k @aledbf @kylos101